MM-HAR: Multi-Modal Human Activity Recognition Using Consumer Smartwatch and Earbuds.

Human Activity Recognition (HAR) is one of the important applications of digital health that helps to track fitness or to avoid sedentary behavior by monitoring daily activities. Due to the growing popularity of consumer wearable devices, smartwatches, and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 44 subjects collected from both in-lab and in-home environments, we demonstrate the limitations of using a single modality as well as the importance of a multi-modal approach. Moreover, we also train and evaluate the performance of five different machine learning classifiers for various combinations of devices such as buds only, watch only, and both. We believe the detailed analyses presented in this paper may serve as a benchmark for the research community to explore and build upon in the future.

Power Optimized Smartwatch-Earbuds Multimodal System for Monitoring Activities of Daily Living.

Activities of daily living is an important entity to monitor for promoting healthy lifestyle for chronic disease patients, children and the healthy population. This paper presents a smartwatch and earbuds inertial sensors based multi-modal power efficient end-to-end mobile system for continuous, passive and accurate detection of broad daily activity classes. We collected various posture, stationary and moving activity data from 40 diverse subjects using earbuds and smartwatch and develop the novel power optimized end-to-end operational system consisting of i) optimized device sampling rates and Bluetooth packet transfer rates, ii) data buffering mechanism, iii) background services, and iv) optimized model size, and demonstrating 93% macro recall score in detecting various activities. Our power optimized solution uses 80%, 40% and 33.33% less battery power for the smartphone, smartwatch, and earbuds respectively, compared to a power agnostic system with an estimated continuous no-charging run time of 50 hours, 16.67 hours, and 25 hours for the smartphone, smartwatch, and earbuds respectively.Clinical relevance- The end-to-end power optimized activity detection system presented in this paper will assist practicing clinicians toward treatment of various chronic disease patients (e.g. diabetes, hypertension, heart disease and obesity) by long-term, continuous monitoring of their lifestyle and sedentary behavior.

Deep Multivariate Domain Translation for Device Invariant Pulmonary Patient Identification from Cough and Speech Sounds.

Pulmonary audio sensing from cough and speech sounds in commodity mobile and wearable devices is increasingly used for remote pulmonary patient monitoring, home healthcare, and automated disease analysis. Patient identification is important for such applications to ensure system accuracy and integrity, and thus avoiding errors and misdiagnosis. Widespread usage and deployment of such patient identification models across various devices are challenging due to domain shift of acoustic features because of device heterogeneity. Because of this phenomenon, a patient identification model developed using audio data collected with one type of device is not usable when deployed in another type of device, which is a concern for model portability and general usability. This paper presents a framework utilizing a multivariate deep neural network regressor as a feature translator between source device and target device domains to reduce the effect of domain shift for better model portability. Extensive and empirical experiments of our translation framework consisting of two different human sound (speech and cough) based pulmonary patient identification tasks using audio data collected from 91 real patients demonstrate that it can recover up to 64.8% of lost accuracy due to domain shift across two common and widely used mobile and wearable devices: smartphone and smartwatch. Clinical Relevance- The methods presented in this paper will enable efficient and easy portability of pulmonary patient identification models from cough and speech across various mobile and wearable devices used by a patient.

Device Invariant Deep Neural Networks for Pulmonary Audio Event Detection Across Mobile and Wearable Devices.

Mobile and wearable devices are being increasingly used for developing audio based machine learning models to infer pulmonary health, exacerbation and activity. A major challenge to widespread usage and deployment of such pulmonary health monitoring audio models is to maintain accuracy and robustness across a variety of commodity devices, due to the effect of device heterogeneity. Because of this phenomenon, pulmonary audio models developed with data from one type of device perform poorly when deployed on another type of device. In this work, we propose a framework incorporating feature normalization across individual frequency bins and combining task specific deep neural networks for model invariance across devices for pulmonary event detection. Our empirical and extensive experiments with data from 131 real pulmonary patients and healthy controls show that our framework can recover up to 163.6% of the accuracy lost due to device heterogeneity for four different pulmonary classification tasks across two broad classification scenarios with two common mobile and wearable devices: smartphone and smartwatch.Clinical relevance- The methods presented in this paper will enable efficient and easy portability of clinician recommended pulmonary audio event detection and analytic models across various mobile and wearable devices used by a patient.